Repository navigation
Add agent readiness section - #1926
nate-double-u wants to merge 3 commits into
Conversation
|
/cc @caniszczyk |
|
Great addition @nate-double-u |
There was a problem hiding this comment.
Pull request overview
Adds a new Documentation / Agent readiness check to CLOMonitor, validating that projects publish an llms.txt or llms-full.txt index either on the project website root (rejecting HTML/SPA fallback) or in the repo (/ or docs/). This integrates the new check across the core linter, scoring, API output, database stats/exports, web UI, and docs.
Changes:
- Introduces a new core linter check (
agent_readiness) with website + repository probing logic and tests. - Wires the new check through scoring weights, report structs/templates, database stats and CSV export, and UI option metadata.
- Documents the new check and updates golden outputs/tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/types.ts | Adds AgentReadiness report option ID. |
| web/src/data.tsx | Adds UI metadata (icon/label/legend/link) and categorization for the new check. |
| docs/checks.md | Documents the new check behavior, IDs, and probing rules. |
| database/tests/functions/stats/get_stats.sql | Updates stats test fixtures to include agent_readiness. |
| database/tests/functions/repositories/get_repositories_with_checks.sql | Updates CSV export test fixture to include the new column. |
| database/migrations/functions/stats/get_stats.sql | Adds agent_readiness to passing-check stats output. |
| database/migrations/functions/repositories/get_repositories_with_checks.sql | Adds agent_readiness column to repositories-with-checks CSV function. |
| clomonitor-linter/src/testdata/display.golden | Updates expected CLI table output to include the new row. |
| clomonitor-linter/src/table.rs | Adds Documentation / Agent readiness row in CLI display and updates tests. |
| clomonitor-core/src/testdata/agent-readiness/llms.txt | Adds testdata file used for repo-based detection tests. |
| clomonitor-core/src/score/mod.rs | Updates scoring tests/weights to account for the new documentation check. |
| clomonitor-core/src/linter/report.rs | Extends Documentation report section with agent_readiness. |
| clomonitor-core/src/linter/mod.rs | Runs the new async check and populates it into the report. |
| clomonitor-core/src/linter/checks/mod.rs | Registers the new check module/config. |
| clomonitor-core/src/linter/checks/agent_readiness.rs | Implements the new check + tests (website fetch + repo glob search). |
| clomonitor-apiserver/templates/repository-report.md | Adds the new check to rendered repository report markdown. |
| clomonitor-apiserver/src/testdata/repository-report.golden.md | Updates golden markdown output to include the new check. |
| clomonitor-apiserver/src/router.rs | Updates API router tests’ report fixtures to include agent_readiness. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
So this is a great start, what about actually just running https://afdocs.dev and create a new top level section under documentation (or above) called "Agent Readiness" and basically puts this in a friendly fashion: npx afdocs check https://docs.example.com --format scorecard |
|
That was going to be next, I can do it up as a part of this PR. |
|
@nate-double-u let's start simple here and wait for @tegioz to review |
|
Sounds good, I'll hold off till we hear from Sergio. |
|
Thanks @nate-double-u. There are some fixes and improvements we'd like to apply to this PR (we'll take care of them ourselves if we continue with this approach). Based on the issues linked, I'm not sure if this is really the best approach for this check. As it is, the check is a bit shallow, checking only for the existence of the Another way of doing this would be to rely on the The alternative to relying on afdocs would be to keep building on top of this PR and implement all checks needed. But at some point we'd essentially be creating another implementation of afdocs that we'd need to maintain as the spec evolves. IMHO, I'd go with Before sending a PR, it'd be great if we could discuss options first. Reviewing PRs takes time, sometimes a lot if they're large 😅. Often, the best way to help is to provide details about what you'd like to do or have, and point out possible solutions like the spec and tool mentioned above. That saves us time and helps a lot. Once we have that information, it's usually easier for us to steer agents ourselves from the beginning, at plan time. Going back and forth with feedback that then needs to be passed on to your agent makes the process slower: we need to review the changes repeatedly, and agents sometimes rewrite large parts of a PR in response to feedback. It also means switching context back to the same task multiple times, often across different days. We really appreciate contributors wanting to help, and the biggest help is often making it easier for maintainers to focus their time where it can have the most impact 🙂 |
Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nate W <natew@cncf.io>
Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nate W <natew@cncf.io>
Replace the native llms.txt check with an Agent readiness report section wrapping afdocs (https://afdocs.dev), one check per spec category, mirroring how OpenSSF Scorecard checks are integrated. The section only applies to repositories with a website configured. Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nate W <natew@cncf.io>
945354e to
c1172df
Compare
|
Thanks @tegioz. I agree about afdocs; I always intended to wrap it, and probably should have left this PR as a draft. I've reworked it to match what you described: the native llms.txt check is gone, and afdocs is wrapped the way scorecard is, with a pinned binary in the tracker/linter images, one run per repository against its configured website, and a new top-level Agent readiness section with seven checks, one per spec category: content discoverability, markdown availability, content structure, page size, URL stability, observability, and authentication. I didn't add any new UI; the new check details reuse the existing hover dropdown. |
|
New screenshots:
|
There was a problem hiding this comment.
🟡 Changes recommended
Unvalidated crawling introduces SSRF risk, while several surfaces mishandle inapplicable categories and AFDocs error semantics.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 51/51 changed files
- Comments generated: 8
- Review effort level: Balanced
| let output = Command::new("afdocs") | ||
| .arg("check") | ||
| .arg(website_url) | ||
| .arg("--format") | ||
| .arg("json") |
| | Category | Score | | ||
| | :-------------- | -----------------------------------------------: | | ||
| | Documentation | {{ category_score(score.documentation) }} | | ||
| | Agent Readiness | {{ category_score(score.agent_readiness) }} | |
| let all_skipped = results.iter().all(|r| r.status == AfdocsCheckStatus::Skip); | ||
| let some_failed = results.iter().any(|r| { | ||
| matches!(r.status, AfdocsCheckStatus::Fail | AfdocsCheckStatus::Error) | ||
| }); |
| .add_row(vec![ | ||
| cell_entry("Agent readiness"), | ||
| cell_score(score.agent_readiness), | ||
| ]) |
| 'agent_readiness', json_build_object( | ||
| 'authentication', repositories_passing_check(p_foundation, 'agent_readiness', 'authentication'), |
| <CategoryProgressbar | ||
| name="Agent Readiness" | ||
| value={props.score.agent_readiness} | ||
| icon={CATEGORY_ICONS[ScoreType.AgentReadiness]} | ||
| bigSize={props.bigSize} | ||
| linkTo={activeLink ? `${props.repoName}_${ScoreType.AgentReadiness}` : undefined} | ||
| scrollIntoView={activeLink ? props.scrollIntoView : undefined} | ||
| /> |
| <th scope="col" className="d-none d-md-table-cell text-center text-nowrap"> | ||
| <small className={`position-relative ${styles.icon}`}>{CATEGORY_ICONS[ScoreType.AgentReadiness]}</small> | ||
| <span className="d-none d-xl-inline-block ms-1 ms-xl-2">Agent Readiness</span> | ||
| </th> |
| icon={CATEGORY_ICONS[ScoreType.Documentation]} | ||
| value={props.data.documentation} | ||
| /> | ||
| {!isUndefined(props.data.agent_readiness) && ( |
|
Thanks @nate-double-u. As I mentioned in my last comment, we'd rather take care of this ourselves. |
|
This is definitely more in line with what I was looking for @nate-double-u, thank you for getting this work started. @tegioz if we can find a way to add this functionality it would be great |
|
Sure, we'll take care of it 👍 |
|
Agent readiness section ready, we'll deploy it next week. Closing this PR in favor of #1931 |
|
Deployed! (Cilium sample results) The new checks are part of the community check set. By default, we use the homepage url set up in the GitHub repository. In some cases, that url may point to a general website instead of the documentation. When that happens, projects can override the url we check from the The checks are only advisory for now, so they don't affect global scores yet. This way we can test everything properly and give projects a bit of time to catch up. clomonitor/docs/metadata/.clomonitor.yml Lines 5 to 12 in 3114ef9 |
Thanks @tegioz! |
Adds an Agent readiness report section that evaluates how well AI coding agents can discover and consume a project's documentation.
Per review feedback, this wraps AFDocs, the reference implementation of the Agent-Friendly Documentation Spec, the same way CLOMonitor wraps OpenSSF Scorecard for security checks. The earlier native
llms.txtcheck is gone. Inspired by linuxfoundation/insights#1830.Open questions
afdocspinned at 0.20.0 in the Dockerfiles, like Scorecard. Is there a bump policy for clomonitor?How it works
afdocs check <website> --format jsonruns against the website configured in the GitHub repository, concurrently with Scorecard (afdocs doesn't use the GitHub token).community+docs.code-only repos) → no crawl, and the section is not applicable and hidden everywhere (score, UI, reports, stats).Breaking changes
afdocsbinary in PATH when the selected check sets include any agent readiness check (community/docs). The official linter container image includes it; CI users running the raw binary with those check sets will need tonpm install -g afdocs./data/repositories.csv) gains 7 new check columns.Testing
cargo test234 ✓ · web 200 ✓ · pgTAP 50 ✓ · fmt/clippy/prettier/eslint cleanScreenshots